Generating Coordinated Natural Language and 3D Animations for Complex Spatial Explanations

Dynamically providing students with clear explanations of complex spatial concepts is critical for a broad range of knowledge-based educational and training systems. This calls for a realtime solution that can dynamically create 3D animated explanations that artfully integrate well-chosen speech. We present a visuo-linguistic framework for generating multimedia spatial explanations combining 3D animation and speech that complement one another. Because 3D animation planners require spatial knowledge in a geometric form and natural language generators require spatial knowledge in a linguistic form, a realtime multimedia planner interposed between the visual and linguistic components can serve as a mediator. This framework has been implemented in CINESPEAK, a multimedia explanation generator consisting of a visuolinguistic mediator, a 3D animation planner, and a realtime natural language generator with a speech synthesizer. CINESPEAK has been used in conjunction with a prototype 3D learning environment in the domain of physics to generate multimedia explanations of three dimensional electromagnetic fields, forces, and electrical current in realtime.